Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfer

Real-world recognition system often encounters the challenge of unseen labels. To identify such labels, multi-label zero-shot learning (ML-ZSL) focuses on transferring knowledge by a pre-trained textual label embedding (e.g., GloVe). However, methods only exploit single-modal from language model, while ignoring rich semantic information inherent in image-text pairs. Instead, recently developed open-vocabulary (OV) based succeed exploiting pairs object detection, and achieve impressive performance. Inspired success OV-based methods, we propose novel framework, named multi-modal transfer (MKT), for classification. Specifically, our method exploits vision pre-training (VLP) model. facilitate matching ability VLP distillation is employed to guarantee consistency image embeddings, along with prompt tuning further update embeddings. enable multiple objects, simple but effective two-stream module capture both local global features. Extensive experimental results show that significantly outperforms state-of-the-art public benchmark datasets.